Papers with Transformer-based LMs
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on how large language models process and store information related to arithmetic tasks have shown their behavior inconsistent and context-dependent. |
| Approach: | They propose to mechanize the processing of arithmetic queries by a causal mediation framework. |
| Outcome: | The proposed model improves the performance of arithmetic queries with a set of MLP modules. |
Empowering Language Models with Knowledge Graph Reasoning for Open-Domain Question Answering (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing Language Models lack the power to store all required knowledge, resulting in a lack of ability to infer out-of-context knowledge. |
| Approach: | They propose a Knowledge Interaction Layer that can be flexibly plugged into existing Transformer-based LMs to interact with a differentiable Knowledge Graph Reasoning module collaboratively. |
| Outcome: | The proposed model can be plugged into existing Transformer-based LMs to interact with a differentiable Knowledge Graph Reasoning module collaboratively. |
Memory efficiency and resource-rational encoding in sentence processing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that language models need to be constrained in their use of working memory for context, the analogue to human working memory (WM). |
| Approach: | They propose to inject noise into hidden representations of Transformer-based LMs to capture constraint on memory encoding. |
| Outcome: | The proposed model improves alignment with human reading times and makes them more compressed and categorical. |